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INTRODUCTION 



The Proceedings of ike Evaluation Workshop Retreat is the third of a 
series of publications dealing with Title III, E.S.E.A. programs, the first 
two being Design for Tennessee Assessment and Evaluation, of Title III , 
E.S.E.A. and the abstract of that document. This document records the 
formal speeches of the workshop personnel but not the comments from the 
discussions that followed. Each speech is retained in its program order in 
an attempt to recapture the major impact of the programming procedure. 
All articles have been edited by the authors previous to the inclusion in this 
w’ork. Only those by Desmond L. Cook differ from his presentation; how ever, 
his three articles form the basis for his conference remarks. 

The majority of the ninety-three participants were nominated by the 
superintendents of their respective school districts to attend the workshop 
because of their involvement in Title III, E.S.E.A. programs. They repre- 
sented forty-tw o school districts and thirty-six counties from the three grand 
divisions of Tennessee. Others present were employees of the Tennessee 
State Department of Education, consultants, and faculty and staff members 
of Memphis State University. 

The beginning of the workshop established the purpose of the meeting 
and set the climate for maximum acceptance of the events to follow. Subse- 
quent speakers emphasized instructional presentations on evaluation design 
components. These definitive statements are fundamental to an understanding 
that evaluation is a vital aspect of project success. Ample opportunity was 
given in group meetings and in leisure hours for amplifying ideas, resolving 
semantic different, illustrating methods of approaching problem-solving, 
and sharing successes and failures in project management. 

It is hoped that this book will be more than a remembrance of three 
days at Montgomery Bell State Park. It is intended that it be an additional 
resource for attacking problems that confront us in all aspects of educational 
management, whether our involvement be direct or indirect, whether our 
role be substantial or minor. 

From this document, a series of reprints will be forthcoming, primarily 
for selected target populations who will use them in preference to the larger, 
more comprehensive voli me, in conferences with a limited scope. 

Joiin Petry, Editor 
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PREFACE 



The publication in the Spring of 1969 of Design for Tennessee Assess- 
ment and Evaluation of Title III , E.S.E.A. was the result of a developmental 
research effort to provide a tool for educators to use in evaluating projects 
and programs in Tennessee schools. Several regionalized meetings were held 
across the state to give local school personnel an initial exposure to this 
evaluative tool. 

It was evident that there was a need for further consideration beyond 
these brief meetings. Therefore, the Evaluation Workshop Retreat was ar- 
ranged to better meet this need The workshop was an intensive three-day 
institute on evaluation using the Tennessee Evaluation Design as the 
synthesizing structure. 

The workshop was organized around the use of consultants for pre- 
sentation of component topics during general sessions. These were followed 
by small-group work sessions under the leadership of discussion group 
leaders. 

Evaluation of the effectiveness of the workshop showed significant 
positive results in a large majority of the components measured. The 
transcriptions included in this proceedings document can be useful to project 
personnel who have evaluative responsibilities but did not attend the work- 
shop. They can also serve as review and reference sources for readers w ho 
desire a means of follow-up and reinforcement. 

Fred K. Bel lot t . Director 
Evaluation Workshop Retreat 



SYNOPSIS OF WORKSHOP PROCEEDINGS 



This document includes the transcription;, of presentations given by 
the consultants at the Title III Evaluation Workshop at Montgomery Bell 
State Park. What may not be clear to the reader at this time is that these 
presentations served as focal points for small, group discussions which 
took much of the time in the workshop and were one of the most important 
parts of the workshop. In these small groups, educators from across the state 
discussed with consultants the issues presented in the speeches. These small 
group sessions were geared to the actual, “at home'’ problems of evaluation. 
Of primary concern for the small groups were the w riting of behavioral ob- 
jectives and the systematic collection of data. 

Dr. Colmey’s overview of evaluation and Mr. Jones' plea for a team 
effort in evaluation set the stage for discussions of these aspects of eval- 
uation. With the move at the national level toward multi dimensional 
criterion measures, more frequent use of observation techniques, imple 
mentation of conclusion-oriented evaluation, and above all, process evalua- 
tion, the opening address was particularly appropriate. 

The need to consider evaluations from the conception of the project 
through the final evaluation was stressed by Dr. Bellott in his presentation 
of the Tennessee Evaluation Design. For those who were so inclined, the 
comparative relationship of the Tennessee Evaluation Design and the EPIC 
Model could be recognized after Dr. Cornell’s introductory remarks. The 
major points stressed in both presentations were the need for preliminary 
planning in the deteimination of goals, the need for communication through 
objectives during the project, ard the need for varied types of observations 
during the operational phase of the project. 

Di .Cook offered help in the administrative problems of project manage- 
ment and Dr. Cornell stressed the writing of bchaviozal objectives from two 
viewpoints. Both, however, pointed to the need for the educator, as a strat- 
egist, to accurately define what types of behaviors will be required of the 
learner in specified situations. Much time was spent in the small discussion 
groups reacting to the presentations about the logic and technique of writing 
behavioral objectives. From the reactions of the participants and the per- 
centage of time spent on the subject, the writing of objectives must be ranked 
as one of the most important topics of the workshop. 

The other speeches by Drs. Nowell, Bowman, and Williams dealt with 
types of data, methods of data collection, and statistical methods for analyz- 
ing data. These topics, of a more technical nature, were also considered in the 
small groups because they relate to the patterns and uses of behavioral ob- 
jectives and evaluation. The relationships among the objectives, measure- 
ment techniques used, and selection of appropriate analyses were stressed, 
providing appropriate points on which to end the workshop. 

Terry Bond, Research Specialist 
Memphis City Schools 
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EVALUATION: AN OVERVIEW 



James W. Colmey, Director 
Bureau of Educational Research and Services 
Memphis State University 



Jome general remarks may give you a little better understanding of why 
we are here and what this Evaluation Workshop should accomplish. What 
are we here to talk about? We know it is “evaluation/ but let us probe a little 
further to see what this “monster" called “evaluation" really is. 

There is a tendency to take old terms that we have used and to change 
them into new' terms or to redefine them. In reality, we ^re really working 
with the same kinds of things that we have done through the years but with 
considerable improvements and modifications. 

We are talking about descriptive data. How do you describe the program 
that you are conducting? That is essentially w’hat it is all about. How do 
we describe something— we observe, collect, write, define, r.easme, manip- 
ulate, present, and analyze. Obviously, we do some other things, but es- 
sentially these just plain words say what it is all about. 

Observation can be a very simple thing. For example, I am observing 
you and you are observ' -g me. But observation through rigid logic, or dis- 
ciplined by controls and cross references, becomes something a great deal 
more complex. In reeiity, we are doing the obvious things which we have 
done for years, but we are becoming a great deal more accurate and objective. 

The men that are on your program will make presentations that will 
extend each of these words that I have casually mentioned, and I am quite 
sure that you will experience some anxiety, confusion and concern about 
what they have to say. First, some words will be used differently from the 
way you use them traditionally. Some words you may not have used before. 
In addition to this, there are some techniques that you may not have used. 
This does not mean that these techniques are entirely different from what 
you have been doing— they are just modifications of what y have been 
doing in your everyday activities as teacher, supervisor, or superintendent 
through the years. 

In the next recade, we are going to depend upon specialists (such as 
the men on your program) who have- worked hard enough on particular 
techniques that they can present them with clarity to large groups (such 
as this one). In today’s world, new knowledge must be disseminated more 
rapidly than in previous decades. What you learn in these three days, you 
are going to have to turn around and teach to somebody else. Charges can 
and must take place within five or ten years that forme* ly took fifty years. 

When talking about new systems, approaches, modifications, or termi- 
nology, we have a tendency to forget the importance of existing documents. 
Do not take weak substitutes for important documents that have been 
available to you through loca* school systems and the Tennessee State 
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Department of Education in the past decade, such as Tennessee’s Annual 
Statistical Report, These documents are a vital part of your evaluating 
system. These are basic to the evaluative formats that we ar^ trying to 
project and evolve into a more advanced form. 

Who does the evaluating? There is nothing exclusive about an evalua- 
tive process and Title III, E.S.E.A. Because there has been more emphasis 
on evaluation for Title III projects, there sometimes is a unique identity 
for evaluation in these projects. Efforts to improve instruction are going 
to increase the need for local, state, regional, and national evaluations in 
all educational programs. There is no monopoly on who evaluates or what 
is evaluated. However, each of these different segments of education will 
require unique adaptations. You will also find that there will be a greater 
need to try to coordinate the activities in evaluation at the local level with 
the kinds of activities that are going on at the state level. At the local level 
you are going to be looking for some different things than at the state lev el, 
but you must have some common ground in order to have both of these 
needs fulfilled. So, in plain language, we will all be evaluating in different 
ways. 

Why do we evaluate? There is only one real good reason. There are 
other reasons that are not so important. We need to have more meaningful 
information to improve instructional programs and to improve education 
for boys and girls in the state. Other things must in some way be related 
i or else they are rather meaningless. 

In terms of objective evaluation with real meaningful purposes, efforts 
; should not be to determine if this is “good” or “bad”— if it is “bad” throw it 

out; if it is ‘"good” keep it. It is not that simple. It is not a meritorious 
j kind of thing where “you get a blue ribbon,” "you get a red ribbon,” “you 

get a white ribbon.” Hopefully, we have gone past that; hopefully, we can 
deal more objectively and meet what tht public is demanding at this point. 

Our general citizenry, legislators, teachers, and children are demand- 
ing improved educational programs. The main reason is that teachers today 
are better educated than they used to be and so are our citizens, legislators 
end children. We have had traditional jokes about incompetent state legis- 
lators— and certainly we knew cases where they have been true and during 
some periods in history where collectively it may have been pretty close 
to tru:. But I assure you that we have legislators in Nashville toda> who 
know T just as much about education as most of us. They are "hardheaded” 
and they want answers because they know that, when they go back and 
talk to people in their community, they arc going to have to talk to people 
who are more intelligent and are looking mere objectively for the kind of 
answers that they want than used to be the case. It is just that simple. 

I used to think, when I was a young school administrator twenty years 
ago, that if I could just get intelligent, well-educated teachers, I would not 
have any more problems. That was one of the biggest mistakes I have 
made. The more intelligent and educated your teachers and students, the 
more insight and questions they have. It does not take them long to call 
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your attention to problems and opportunities. Therefore, you must be 
more straightforward than used to be the ease. Within the next ten years, 
this will be a much more important consideration in Tennessee tl jn in the 
past ten years. Fact gives merit to the kind of conference that we hope to 
have in the next few days. 

Next question, how do we go about this process? The men on your 
program are going to answer that question, but I can give you a few gen- 
eralities that will be the background lor their remarks. First, they will be 
talking about evaluating as a never ending series of assessments, measure- 
ments, and analyses. Participation is all going to become more complex, 
useful, and objective. There will be more frustrations in this process, but, 
presumably, better decisions can be made because of it. 

Some educators say that we have an infinite number of variables to 
deal with; therefore, research discipline does not mean anything if you 
try to be objective. Some will carry clinical research concepts to the point 
of having no utility in conclusions. Either extreme is unrealistic. The fact 
that educational research and evaluation cannot be as precise as physical 
science does not mean that education can not improve itself through an 
orderly process of observation, measurement, and analysis. We hav e got to 
believe that. 

A weak spot in educational evaluation is that many components are 
appropriately accomplished but frequently left unrelated. We have a ten- 
dency to work in isolation. One group does testing, another group writes 
objectives, another group makes decisions. We do not always put these 
efforts together, and we do not always get the benefits from a strong con- 
tribution. In the next year or two, your state and local evaluation activities 
should be more closely coordinated. Tennessee has had a better state-local 
relationship than I have observed around the country. 1 think it is fair to 
say that in the last few' years this relationship has improved. We have had 
more opportunities to work together. It is appropriate that universities 
participate and contribute, and it is appropriate that other agencies of the 
state, such as welfare, begin to work with state and local educators. 

In the next three days we will be talking about tools for management, 
some improved tools that have been available to us and some new T tools. 
We aro trying to demonstrate new dimensions. The only reason for doing 
this, if we are honest and sincere, is to improve education for boys and 
girls. If you keep these things in mind and if you recognize that the pre- 
senters of material at this workshop are specialists who are going to bring 
to you specific kinds of information, you will benefit more from this work- 
shop. The presenters are searching for w ays to help us, and what they say 
will have moie meaning to you, if you think of their remarks within the 
context of the comments I have made this morning. 
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THE TENNESSEE EVALUATION DESIGN 



Fred K. Bellott, Associate Director 
Bureau of Educational Research and Services 
Memphis State University 

Thank you, Dr. Colmey. Much of our work is carried out in a team effort 
and members of the team cannot work in isolation. If we did try this, I am 
afraid that we would not get very far. The planning and conducting of this 
workshop was very much a team effort, not only in the Bureau but also 
between the Bureau and the Tennessee State Department of Education and 
all consultant members of the workshop staff. 

I would like to make a few comments in reference to the pre-test that 
we were working with earlier this morning. First, I should explain that this 
was made up from questions submitted by each person who will be making 
a presentation during these three days. From the something like ninety 
questions that were submitted, the pre- and post-tests were selectively com- 
piled. This is the primary reason that th.i vests have a varied format and a 
varied typo of response. We tried not to constrain those people who were 
supplying the questions in using a specified format. They were given com- 
plete freedom in formulating their test items. Another reason was that we 
wanted to demonstrate several formats and the use of different kinds of 
measures. You may have noticed that the last area in the test dealt almost 
exclusively with affective behavior while the others were objectively 
oriented to cognitive behaviors. The last section also illustrates a semantic 
differential instrument which utilized a Likert-type scale as the means of 
response. 

There will be a post-test using a similar instrument on Wednesday. 
It is from the comparison of the two that we would be able to draw some 
conclusions about our success in this three-day workshop. Again, with ref- 
erence to the pre-post design, we are evaluating the staff, not you, the 
participants. 

I would like to build upon several things that Dr. Colmey has said in 
terms of a frame of reference for the Tennessee Evaluation Design. First 
of all, I think that we should recognize the significance that is attached 
to the name of the model we will be working with in this workshop- -The 
Tennessee Evaluation Design. It has been submitted to, and some affirm- 
ative endorsement has been received from, the Tennessee State Department 
of Education as a design that is appropriate for use throughout the state 
with most projects that are utilizing evaluation procedures. The design 
itself relates quite closely to several others with which you may be familiar. 
The scientific method provides one such comparison. Jack, I remember 
your mentioning this earlier in our discussions: there is a close parallel 
here between the step p.ocedures found in the evaluation design and what 
we have recognized as the scientific method. If you are oriented to data 
processing and some of the procedures utilized, you might think of this as 
more closely related to the systems approach. Again, in the step by step 
sequence it is quite closely related. 



6 



13 



The Tennessee Evaluation Design has four phases. We have called them 
Status evaluation, Planning evaluation, Operational evaluation, and Final 
evaluation. They are similar in nature. Earlier this year, the State Depart- 
ment, along with the State Testirg Bureau, and the Bureau of Educational 
Research and Services, disseminated an instrument to gather information 
for a “state wide assessment of educational needs." The assessment instru- 
ment went to a sampling population of about 4,800 randomly selected 
teachers, plus all principals, counselors, and superintendents in the state. 
This was the first move toward Status evaluation for the state on a strati- 
fied basis. Status evaluation demands the definition of the operational con- 
text to determine what it is we are working with, and where we are. It should 
sen e somewhat the same purpose as our pre-test this morning — to establish 
the pre-existing conditions. 

From this we move toward the identification of needs. If we had in fact 
made a proper type of assessment today, it should have been reflected in our 
program. The program would presumably be designed to meet the needs 
which had been identified. 

Using the data collected in the assessment and also utilizing other exist- 
ing data, such as in the annual statistical reports and other kinds of resources 
for what I call “base-line" data, we should identify what needs exist. These 
base-line data (Dr. Colmey earlier introduced this element), we feel, are very 
important. There are many areas wherein we do not have base-line data 
currently existing. Where this situation does exist we need to build into our 
Status evaluation the collection of base-line data; but. where these data al- 
ready exist, there is not a reason in the world why we should not use them. 
Sometimes, I am afraid that we frequently go through the exercise of re- 
discovering the wheel simply because we do not go to the trouble to find out 
what has already been done. If wc are asking twenty items that were just 
asked a month ago, or six months ago, or last year, the data is still valid 
data if the situation has not changed since that time. The description of the 
context remains the same and, where these situations exist, I certainly en- 
courage the use of ex.sting base-line data without going back and getting it 
again. Again, our problem in Status evaluation is to determine where we are 
and what our problems are. 

We move from that into Phase II, or Planning evaluation. Once we have 
identified needs, what are we going to do about them? What do we do? It 
is at this point that our objectives begin to emerge. In order to deal with 
the goals that we would define here, we certainly need to look at several ways 
of attaining these goals and how- we would define the objectives. Then we 
have to consider the alternatives that are available to us, not only in terms 
of a general design but also in terms of a specific strategy. Many times we 
belatedly find ourselves in a position of having to make evaluation after the 
fact. For example, someone who has been operating a project for two years 
suddenly realizes, "we have got to evaluate.” What does one do in a week's 
time to get an evaluation made? 

We feel that this is an inappropriate approach to the task of evaluation. 
Evaluation ought not be imposed after the fact; it ought to be included at 
the beginning of a program or project. At the time the objectives are estab* 
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lished, we should have a v clear idea about what other activities we are 
go ng to evaluate and h u. will be done. We may want to use a pre-post 
design. If the decision is made after the fact, it may be that the only thing 
one can do is to take a post-test. But if we do not know' whe^e we were to 
start with, how' do we measure gain? This is not to say that the use of the 
post-test design is necessarily inappropriate. There are other applications of 
design that can yield much more information to us if we have the use of fore- 
sight instead of hindsight. 

Planning evaluation includes thinking through these types of consider- 
ations. We need to determine the stiategy of problem solution and the design 
that is to be used in operational evaluation— the means of measurement. Are 
we going to use standardized tests as the only type of object ive measure- 
ment? Will we utilize subjective measurements taken in process? These 
determinations should be made before the project gets “down to the wire.” 

Let us move now- to Phase III, Operational evaluation. We have our 
project underway and we are evaluating 0 use the term as a description of 
our continuous evaluative activities}. We are not only aware of what we are 
going to do, but we are in the process of doing it— because Operational 
evaluation is a process, a continual process. It is not a “one-shot” approach. 
From the time your project becomes operational, you have several tasks 
in developing and/or selecting the instruments that will be used for measure- 
ment and periodic observations. 

Dr. Colmey dwelt on the term “observation” for a while and I would 
like to expand that a little. We like to think of the term “observation” in 
a broadened context, not as just looking at something but as a recorded 
description. I can look at Terry (Dr. Cornell) and, as I observe him, I may 
or may not go through an evaluative process subjectively in my mind. I 
consider that the three letters I received from him this past month are 
sources for observations, I hope to have additional correspondence and 
these will enable additional observations of him to be made. These examples 
are personal observations which Dr. Cornell and I have made recently. If 
we were asked to evaluate something to which these observations were rele- 
vant, they would be useful. 

So let us use the term broadly, not just limit its application to one kind 
of operation. An observation may be in the form of the pre-test that we had 
this morning; it may be interpreted simply as our looking at one another. 
But an observation is an important element of Phase III, Operational eval- 
uation. 

From the observations that we have made, presumably (if this is to have 
real meaning to us) we will have to have some type of analysis. We w'il! have 
a record of what has transpired during this project which w ill include all of 
the observations made. But the analysis itself is another step in Phase III. 
Analysis means to break down and look at the operation microscopically, to 
separate it into its parts. If you are not going to separate the j urts, you 
simply get a total spectrum of the whole and cannot see it analytically. 

After the analysis of the data, though, we must synthesize the infor- 
mation and consider it in the context from which it came. In the same way 



that the references were made earlier to “not working in isolation/' we have 
to put these data together as emphasized so that they give us meaningful 
information about our total effort, our total project. In this way, we can have 
a feedback of information from the analyses for the purpose of decision* 
making. If we have analyzed scores, for example, on performance in a read- 
ing project and have found that we are not approaching our objective, then 
we feed this information back into the cycle. We can make modifications in 
the project at that time. We would do the same thing with our evaluation 
in any project, if we find that we were not getting the kinds of measurements 
that we had anticipated. On our recycle we have the advantage of the feed- 
back. 

This is a matter of communication, and so often this is where our own 
organizations begin to come apart at the seams. We do not practice what we 
preach in the use of feedback. The administrator does not wait for the feed- 
back of information before he makes a decision; or, the testing consultant, 
after he has received his data and analyzed it, lets it stay in the drawer. 
These kinds of blockages continually harass and prevent us from getting the 
whole evaluation process into operation in a meaningful manner. 

Phase IV is Final evaluation. The label used here, Final evaluation, was 
purposefully chosen- We are looking now at the whole project. We have a 
series of observations because we have monitored the project in process. 
Now it is time to look at the total effort and see what it all means. We were 
collecting data, then, for the total p oject. We will analyze the data con- 
cerning our pre-port status, or change measures however they were deter- 
mined. Then we can interpret these data in terms of the criteria that tee had 
previously established. These are the criteria that were set forth in Phase 
II in our objectives. They mRy have been modified since being written down 
originally, but they represent the kind of achievement that we have expected. 
These criteria (that have been determined already) should be used in inter- 
preting the data and arriving at a point of decision. From these we si mid 
have a recommendation about the next cycle or to the administrator who has 
to make a program decision. 

It is my feeling that information from evaluation ought to be conclusion- 
oriented. It ought to enable us to determine an appropriate course of action 
— whether what we have done was a "good way of doing" or whether the 
ways in which we previously attacked the problem were better, measured 
by how much success we attained in it. It also ought to enable the adminis- 
trator to make a decision that would justify our support and our money 
or other resources. Is this the best alternative? If the evaluation is not 
decision-oriented, in my opinion, it has questionable value. This has appli- 
cation to the local level as well as state and national levels. I think that, if 
we cannot provide evaluative information to the State Department by which 
they can make program decisions, we have no right to expect them to decide 
in favor of our program. The same kind of application, of course, applies to 
your local school boards. 

The use of the information that comes from this evaluative process, I 
think, is the most important element in tho- whole design. If you do not use 
it, you should not have gone to the trouble to gather it. It has to become 
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meaningful by affecting the program development and the decisions that are 
made for support of them, and for the development of further programs or 
the broader scale of implementation. 

I have tried to sketch through the design and its four phases and the 
components with which we will be working and will deal with in detail in 
forthcoming sessions. During the afternoon session, Dr. Desmond Cook will 
be with us. Dr. Cook has worked with planning in a particular frame of ref- 
erence. that of management. Some of his techniques have been utilized in our 
own projects in development. 
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BETTER PROJECT PLANNING AND CONTROL 
THROUGH THE USE OF SYSTEM ANALYSIS AND 
MANAGEMENT TECHNIQUES* 



It was my good fortune this August to have been able to attend a five- 
day course at UCLA focusing upon a critical appraisal or state-of-the-art 
review of management information systems as they are now developing in 
the area of our society most commonly referred to as the military-industrial 
complex. The forces of fate, or just some simple clerical operation, resulted 
in my roommate being a technical representative ( i.e. , salesman) from one of 
the leading dat u processing hardware manufacturers in the country. During 
the course of a heated session (i.e., the room temperature was up and the 
speaker boring), my roommate uttered an appraisal of the program by 
stating that while he was not learning much new at least the speakers and 
participants were using the right set of “buzz” -words in thtir presentations, 
discussions, and conversations. In my best academic manner so that any 
naivete and innocence would not be betrayed, 1 cautiously inquired as to 
what he meant. My companion replied that in almost any professional field 
there is always a current set of terms that one must use in his speeches, cock* 
tail conversations, and business dealings to show that he is with it or in. It 
was comforting to me to learn that fields other than education have a prob- 
lem similar to the one with which we are faced. 

Education is beset with its own set of "buzz” words plus those from out- 
side of education. The list of such words is relatively long and hence 1 should 
not use the time allocated to me to cover all of them. For those of you not 
familiar with some of the terms and/or the ’‘buzz” word game, I would call 
your attention to an opportunity of being initiated which is presented in the 
September issue of the Phi Delta Kcppan* wherein there is an opportunity 
to devise your own set of buzz w’or^s on a somewhat random basis. 

I am sure that the symposium which you «re attending has already pre- 
sented, and will continue to present, its contribution to the pool of “buzz” 
words. The purpose of the symposium is not, however, to introduce new 
words into the language of the educational situation. Instead, it is a recog- 
nition of the fact that certain words have already become part of (he 



‘A paper presented at the Symposium on Operations Analysis of Education, 
sponsored by (he National Center for Education Statistics, U.S. Office of Edu- 
cation, November 20*22, 1967, Washington, D. C. Appreciation is expressed to 
Donald Miller, Roger Kaufman, Ed Novak, and Duane Oilman for their many 
helpful comments and suggestions, 

•■'Naming 1,000 Educational Innovations," Phi Delta Kappan (September, 
19671,36. 
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educational language. It is necessary, therefore, to give some attention 
to both the clarification of such words and to the possible role that the 
realities represented by the abstract word might have in the present and 
future of education. 

In this paper, I propose to deal only with a selected set of “buzz” 
words derived from the more general one called Operations Analysis (work- 
ing on the perhaps tenuous assumption that it is the generic term). The 
specific words that I would like to discuss are those of project planning pjid 
control system analysis, and management techniques. My single purpose 
is to indicate how these concepts can be related to and used in an activity 
which has been and is consuming more and more energies of professional 
educators— the activity is the preparation end execution of research and 
development projects. 

DEFINING PROJECT PLANNING AND CONTROL 

What is a project In view of a lack of consensus about a single defi- 
nition, Gaddis provides a useful one for our purpose: 

A project is an organization unit dedicated to the attainment of 
a goal— generally the successful completion of a development pro- 
duct on time, within budget, and in conformance with predeter- 
mined performance specifications . 4 

While the above quotation provides the essential description of a project, 
the definition presented has a limitation for our use since it only relates to 
development projects. I should like to expand the definition to include the 
broader spectrum of research and engineering efforts as well as product 
development activities. 

Projects of the above types have several common characteristics or 
features . 1 In general, projects can be said to be finite , or having a definite 
end point; complex, in that a mix of human and material resources is used 
to do a series of linear or parallel-related jobs; homogeneous, in the sense 
that one project can be marked off from another project or from the environ- 
ment within which it exists (we will return to this characteristic later in this 
paper); and nonrepet it ive, in that it is usually a one-time effort and there- 
fore often has some uncertainty associated with it. 

Planning is used here in its most general sense as described by Emery •, 
that of outlining the future and/or deciding in advance what is to be done. 



J Much of that which can he said here about projects applies also to pro- 
grams. Major differences might be in scope, magnitude and duration. 

*PaulO. Gaddis, "The Project Manager,” Harvard Business Review <May, 
1959), 89-97. 

‘IBM Application Description, Project Management System 360. Program 
Application H 20-0210-0, 1966. 

•James C. Emery. “The Planning Process and Its Formalization in Com- 

f jter Models,” Second Congress on the Information System Science, ed. by 
oseph Spiegel and Donald Walker ( Washington, D. C.: Spartan Books), 1965. 



2 0 13 



The output of the planning process is a plan which may take the form of a 
budget, rules, programs, schedules, and similar items. As Emery notes, the 
purpose of a plan is to bring about behavior that leads to desired outcomes. 
To accomplish the latter, the plan must (a) describe actions (some synonyms 
are procedures t process descriptions, and activity specifications) and out- 
comes ( some synonyms are declaratives, state descriptions, or product speci- 
fications) and lb) serve as a formal tool or vehicle for management. 

Control is used here in its most general sense of a monitoring function 
to make sure the plan is being effectively and efficiently carried out. The 
control formula of noting deviations from plan, taking necessary corrective 
actions, and recycling is included within the concept for purposes of our pres- 
ent discussion. 



The concept of system analysis presents some difficulty in being de- 
fined because of some confusion existing in the field at the present time. 
Let me start by noting, first of all, that the term system is being used in 
its singular and not in its plural sense. There is much confusion today in 
education, and perhaps even in other fields, relative to the use of the plural 
“systems” as an adjective and/or noun. Educators talk about systems 
approaches, instructional systems, and even systems analysis, but one is 
not sure whether the singular or plural is being intended. 

System as used here refers primarily to the orderly (i.e., logical) arrange- 
ment of interdependent components or parts into a connected or interrelated 
whole to accomplish a specified goal. So defined, it is assumed that a system 
can be factored or resolved into a series of subsystems and each subsystem 
itself can be further factored or resolved.’ 

For a meaning of analysis, the ideas presented by Starr in his book on 
production management 1 are useful to us. Starr defines analysis in terms 
of the principle of disassembly. Under this principle, analytic behavior con- 
sists of operations that involve division, dissection, classification, parti- 
tioning, and similar actions. The operations of summation, integration, 
unification, and similar actions that relate to a principle of assembly, are the 
act of synthesis. It wiil be reassuring for many of my educational colleagues 
to know that synthesis has elements of Gestalt psychology associated with 
it. 

Combining our concepts of system, analysis, and synthesis we can now 
define system analysis as that process of disassembling some objective- 
oriented whole into its component parts. System synthesis consists of 
putting the parts back together again into some kind of a whole. Figure 1 
shows these two concepts in a graphic manner. 



1 David O. Ryans, ' System Analysis in Education,*’ Technical A femo (July), 
1964. 

• Marlin K. Starr, Production Management: Systems and Synthesis (Engle- 
wr) od Cliffs, Ne-v Jersey: Prentice-Hall, Inc., 1964). 
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Figure 1 
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DEFINITION OF MANAGEMENT TECHNIQUE 

Management technique is not quite so difficult to define as the previous 
terms. Any attempt to define what is meant without a prior reference to the 
nature of management would be somewhat premature. Basically , manage- 
ment can be considered as a process which involves the functions of 
planning, organizing, directing, and controlling the personnel and other 
resources needed to accomplish an object ive or goal. It is generally recog- 
nized that a manager's principal rota is to make decisions with regard to 
each of the functions noted above. 

Many techniques (or systems) have been developed in order to make the 
manager’s task of carrying out the above role an easier one. In their recent 
book on management systems, Archibald and Villoria provide a useful defi- 
nition of such a system: 

We may define a management system as a set of operating pro- 
cedures which personnel ca*ry out to acquire needed information 
from appropriate sources, p ocess the data in accordance with a 
pre-programmed rationale, and present them to decision makers 
in a timely, meaningful form. Most contemporary systems involve 
manual data collection and input, machine processing, tabular and 
graphic output production, and human analysis and interpretation. 
Thus we can say that the systems collect, synthesize, process, 
transmit, and display information, which flows from a primary 
source, through an editing, computation, and selection process 
to the manager.* 

Two principal ideas are highlighted in this definition. First, management 
systems are designed to provide infoimafion. Second, the decision-making 
operation is left to the human manager. Some sophisticated systems do have 
what is called pre-programmed decision-making (i.e,, rules) as an inherent 
part of them. Most systems, however, still rely upon the use of humans to 
make what in effect are non-programmed decisions. Most of the systems 



•Russell D. Archibald and Richarc L, Villoria, \ctuork-Bafcd Manage- 
ment Systems (New York: John Wiley and Sons, Inc., 19671. 
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developed are primarily aimed at facilitating the control function of a given 
manager's job. In view of the definition and ideas presented above, it is 
perhaps more appropriate to talk about management information systems 
rather than management systems. There is a general consensus that 
managers operating within the project context require data or information 
relative to time, or schedules; cost, or resources; and performance, relia- 
bility, or quality of objective accomplishment. Of these three types of basic 
data, the most common data obtained and used in a project situation is that 
relating to time or schedule. 

A wide variety of numerous management systems or techniques have 
evolved over the past several decades to facilitate the manager's task. Three 
general types of systems have been developed. One group relates to the 
quality characteristics of a product, a second category relates to the op- 
erations involved in producing the product, while a third group relates to the 
administrations involved in carrying out the operations. The selection of a 
particular technique for a particular situation is not easy because a system 
designed for one purpose may not be suitable for a different purpose. I have 
chosen to delimit my definition of management techniques to that group 
of selected operations— related systems knowm as network techniques which 
have become increasingly popular during the past decade because of their 
relatively high degree of success in carrying out research and development 
activities with the military-industrial complex of our society. The appli- 
cation of such techniques to the field of education is just beginning, but 
their value has already been demonstrated- 10 

To summarize briefly, definitions have been provided for three current 
buzz words from the general area of Operations Analysis in education. Pro- 
ject planning and control was defined in terms of pointing out a concern 
with the planning or outlining of the future for a goal-directed activity which 
is finite, complex, homogeneous, and nonrepetitive in nature, plus moni- 
toring and correcting deviations from the plan. System analysis was de- 
fined in terms of the analytic procedure involved in disassembling a goal- 
oriented whole into its component parts and then re-constituting it or syn- 
thesizing. Management techniques were defined in terms of their role of 
providing managers with information needed to control the accomplishment 
of the project effort w r ithin established time, cost, and performance para- 
meters. 



TOWARD PROJECT IMPROVEMENT 

Having defined the three essential term3 of concern in this paper, albeit 
briefly and simply, let me now present or develop the major point or thesis 
of this paper. It is contended that a sizeable number of projects in the field 
of education have been inadequately planned in their initial development snd 
improperly executed once started because the initiators of such projects 



19 Desmond L, Cook, PERT: Applied ions h Education, Cooperative Re- 
search Monograph No, 17, OE-12024 (Washington, D. C.: U.S. Government 
Printing Office. FS5.212:1202f 1966). 



9 3 



16 



were just not simply aware of the new tools and techniques available to 
them for project planning and control. It is further contended that the com- 
bined applications of system analysis and management techniques would 
be of immense value in producing better planned and controlled educational 
projects than has been the case in the past. 

The basic premise for this position set forth is that the typical research, 
development, or engineering project in education can and should be funda- 
mentally thought of as being a system. Viewing a project as a system can 
be justified primarily on that project characteristic referred to earlier as 
homogeneity. Any project can be visualized as being made up of a series 
of parts, units of activities which belong exclusively to that project. Hence, 
one can speak of the project as being an entity possessing edges and 
boundaries that help to distinguish it from the rest of the environment with- 
in which it operates. One writer put it, . . a project has a greater density 
of dependency within itself than between it and its surroundings.” 11 The 
suggestion is made further that the boundary zone can be approximated 
where there are relatively few' activities leading outward — that is, where 
there seems to be relatively natural perforation. Even though it can be dis- 
tinguished from the rest of the environment, the project still operates within 
an environment which affects it or which the project itself affects. Under 
this definition or characterization, the concepts of system analysis and man- 
agement techniques assume validity and become useful tools for project 
planning and control. The question now' is, “When, where, and how are they 
applicable?" To answer this question, let us first examine the general steps 
in planning and controlling a project. 

The initial step is to establish the goal or objective. The subsequent 
steps are to do a project definition, develop a project plan, and establish a 
schedule. The second step of project definition is essentially a process of 
disassembling or breaking out the many jobs which have to be accomplished 
to reach the stated objective. Putting it another way, we do an analysis of 
the system — or a system analysis. The end product of this definition or 
analysis step usually takes the form of a hierarchical plan or chart showing 
several levels of prime and supporting functions and tasks 11 (i.e., objectives! 
which have to be accomplished in order to accomplish the p,oal of the pro- 
ject. Functions or tasks can be further factored so that the smallest unit for 
planning purposes is established. Project definition through system analysis 
should not be thought of as an easy task. One difficulty is a clear establish- 
ment of that set of hierarchically ordered tasks or functions which helps 
to establish the boundary between the project and the environment within 
which it w ill operate. Another difficulty is the inability to define the measur- 
able goals of the project. If the goal is a product, then the process to be de- 
fined by the analysis to achieve the product becomes quite tenuous if that 
original product is defined loosely. 



n William H. Huggins, ‘'Flow Graph Representation of Systems," Oper- 
ations He scare Pi and Systems Engineering (Baltimore: John Hopkins University 
Press, 1960). 

,r The hierarchical plan could be established using variables other than 
tasks or functions, but the latter ere the ones fairly commonly employed 
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Having defined the project by use of analytic techniques, the project 
plan can now be developed by employing a graphic representation of the 
order in which the many functions or tasks have to be accomplished in order 
to reach the project objective. For this purpose, we can capitalize upon one 
of two methodologies system personnel have developed for representing any 
given system under consideration. One methodology involves the use of 
mathematical models or equations. While highly useful and very sophisti- 
cated to some system analysis, the set of equations has limitations in that 
it fails to portray in a readily comprehended form the structure of a system 
as a whole . 13 The second methodology for a system structure representation 
takes the form of various flow graphs, more commonly called flow charts/' 
"block diagrams*' and similar terms. The purpose of such diagrams is not 
necessarily to portray the things that comprise the system but rather to 
show the various operations that the system performs upon the Ftuff it pro- 
cesses. Huggins has referred to flow graphs as a kind of mathematical 
Esperanto for system analysis, design, and simulation. They have the 
advantage of permitting a human to perceive quickly a total pattern plus the 
relationships existing therein and of being more palatable to the relatively 
unsophisticated person than are mathematical models. 

The flow' graph model which appears to have high relevancy for system 
representation is that one composed of a network of branches that connect 
at nodes or points. Each branch may be thought of as originating at one node 
and terminating at another node with direction from one node to the next 
point being indicated by an arrow. For our immediate purpose, each of these 
branches can be equated to or identified with the functions or tasks which 
must be accomplished to reach the goal. 

Flow' graph methodology permits one to achieve a synthesis of the func- 
tions identified through analysis. This technique helps to assemb’e that 
pattern of relationships, dependencies, and sequences which might be es- 
tablished as the most efficient way of "moving through the project," This 
"moving through the project” refers to the idea expressed earlier that flow' 
graphs show* how' we intend to process the '‘stuff" that goes through the 
system. In a project, this "stuff" is most probably the intellectually-related 
activities required to accomplish the project objective. We are not able to 
portray graphically the intellectual processes involved in any of the functions 
or tasks, but we can show’ how we intend to organize and sequence our think- 
ing and related activities as we move through the project in so far as the 
tasks and functions identified reflect such intellectual or cognitive processes. 

Within system analysis techniques, a "functional flow T diagram" is often 
employed to show the functions to be performed and the flow of these func- 
tions in meeting the objectives which have been identified. 1 * The functional 
flow diagram has limited use is a management tool for making sure that the 



“Huggins, 

14 K. A. Ka’ifman and R, E. Corrigan, What Is The System Approach, and 
What's In It for Administrators (Orange, California: Chapman College, 19071. 
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several functions and tasks are accomplished during the execution phase 
of the total project effort. 

The concept of flow graphs to represent a system is somewhat of a 
recent origin. A very similar idea, however, was advanced by Gantt in the 
development of bar charts during the first World War. 1 * Gantt charts served 
a useful purpose, but they had the limitation of not fully integrating the 
parts of the system into a component whole. Partial analysis was achieved 
through Gantt charts since the diagnostic synthesis perhaps served as a 
forcing function on what was analyzed. The analysis did, however, usually 
miss the interaction relationships. Other techniques using a visual or graphic 
representation of a function or task breakout were developed subsequently 
to the Gantt chart, but time does not permit a discussion of them. The most 
recent innovation has been the conceptualization and development of man- 
agement systems which utilize the system analytic and synthesizing pro- 
cedures combined wdth the flow' graph concept of nodes and branches. The 
general terms for such systems are network techniques; PERT/CPM are 
the most commonly known systems. 

Network systems such as PERT/CPM usually require, as a first step, 
that the project objective be identified. Then the elements of the total pro- 
ject are identified and placed in some type of hierarchical order. This 
essentially analytic procedure results in a product most generally known 
as the "workbreakdown” structure. Within network techniques, the work- 
breakdown structure most often represents the end products (either in the 
form of hardware or software) of tasks or functions rather than a repre- 
sentation of the tasks or functions themselves. There is no reason, however, 
to believe that such a product orientation must always obtain. Workbreak- 
dowm structures can be composed of tasks and functions just as easily as of 
products. In either case, the workbreakdown structure represents the pro- 
ject definition phase of total project planning. Based upon the workbreak- 
dowm structure, a network consisting of activities or functions (our original 
branches) and events (our original nodes) is developed with a uni-direct ional 
flow (usually from left to right). The network integrates the constituent 
elements of the project into their necessary sequence and dependency rela- 
tionships ( interactions) in much the same way as the functional flow r diagram 
does in system analysis. Once the network is established, PERT/CPM have 
provided procedures for securing time estimates for the completion of each 
of the several activities or functions. These activity time estimates are then 
used to derive estimated start-completion times for the start or completion 
of functions, including the final node in the network. Schedules are then 
developed for the accomplishment of events or activities after careful con- 
sideration of resource availability and/or allocation. 

One? the project definition phase involving analysis has been completed, 
the workbreakdown structure prepared, the process of synthesis accom- 
plished through the network or flow’ graph procedure, and the time esti- 
mates/schcdule procedure completed, the essential steps of project planning 



ll Wallace Clark, Th<* Gantt Chart (New York: The Ronald Press, 19221. 
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have been accomplished. The result is a plan foi the future— a graphical 
representation of the project tasks as they are to be accomplished in order 
to achieve the project objective. It should be noted here that we have 
approached the problem as if there were only one way to reach the project 
objective. Actually, there are alternative ways of reaching the goal. Each 
alternative with its associated project definition, project plan, and sched- 
ule is a separate system. If we prepare several alternatives for consider- 
ation, each may be considered as a separate system and thus we are ap- 
proaching the systems analysis concept developed by Hitch and McKean. 1 * 
Preparation of alternative systems to reach the project goal for manage- 
ment decision requires information relative to the associated costs for each 
system. 

The network with its associated schedule provides us with a means of 
managing the project to make sure that the several functions or tasks are 
being completed in proper sequence and that we are staying on our time 
target. If the progress through the project is normal and on schedule, we 
can feel safe in assuming that the initial analysis and synthesis operations, 
plus the time estimating procedure, have been fairly well done. Should we 
begin, however, to observe malfunctions in the system, represented by 
nodes or events not being reached on schedule, or there is frequent modi- 
fication oF the plan in that new' tasks are continuously inserted into the flow r 
graph, such an observation might indicate that there was perhaps a failure 
to do a complete and thorough analysis and synthesis operation in the ini- 
tial development of the project plan. Under this situation, valuable project 
time would b' lost as resources were redeployed to conduct the necessary 
replanring operation. A schema illustrating the suggested relationships 
among >ystem analysis, management, and project planning and control 
steps is presented in Figure 2. The middle box shows the steps in project 
planning and control while the relationships to system analysis are shown 
on the left and to management techniques on the right. The schema points 
up rather clearly that there is a great deal of similarity between the general- 
ized sequence of steps under system analysis and synthesis and the steps 
of workbreakdown structuring and networking under network based man- 
agement systems. Putting the results of the above steps into a time frame 
puts us into the realm of management control systems. 



SUMMARY 

The purpose of this paper has been to show r how two current buzz 
words employed in educational circles— $>\sf em analysis and management 
techniques— might be employed in combination to produce better planning 
and controlling of educational research and development projects. Ex- 
perience with the use uf these techniques has demonstrated that better 



‘*C. J. Hitch end R, N. McKean, T he Economics of Defense in The S’uclcar 
Age (Cambridge?, Massachusetts: Harvard University Press. 1900) 
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planned and controlled projects result than might otherwise be the case. 17 
The employment of network techniques, such as PERT and CPM, for pro* 
ject planning and control, not only have advantages in and of themselves, 
but they also provide a very useful vehicle for an initial orientation into 
some rather complex operations— those of system analysis and synthesis 
procedures and the representation of existing and proposed systems through 
flow graph methodology. 

The temptation to employ just one more buzz word cannot be resisted. 
The essential position in this paper is that the combined use of system 
analysis and management technique will have a synergi'srfc effect upon 
project planning and control. As I understand synergistic, it is the effect 
of the two techniques operating in combination to produce an effect that 
either one would or could not produce by itself. The simple contention of 
this paper is that the synergistic effect from using system analysis and man- 
agement techniques in project planning and control cannot be other than 
beneficial provided that such procedures are fully understood and properly 
applied. While rot curing all ills, they can go a long way in helping with the 
stresses and strains exhibited under a bad case of “projectitis 1 ’ under such 
programs as ESEA Titles I and III. 

Let me close with a quotation which appears to be relevant both to the 
intended tone of this paper and the general topic of the symposium, it is a 
quotation from Alice's Adventures in Wonderland by Lewis Carroll. 

‘Speak English, 1 said the Eaglet, T do not know the meaning of half 
those long words, and what's more 1 don’t believe you do either.’ 

And the Eaglet bent down its head to hide a smile; some of the 
other birds tittered audibly. 11 



11 For example, ihe joint work of the author wilh that of R. Kaufman. B. 
Corrigan, and 1). Miller on the use of systems analysis and synthesis techniques 
and management systems in preparing educational planners for the state of 
California. 

" Lew is Carroll, A Her 's Adventures in IVcru/erfand (New York: St. Mar- 
tin’s Press. 1965). 
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THE NATURE OF PROJECT MANAGEMENT 1 
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INTRODUCTION 

The role or position of project manager, also referred to as project 
director or principal investigator, is a relatively new role not only in the 
field of education but also in other areas, such as the military, government, 
and business- No specific point marks the introduction of this role. It has 
been gradually emerging in the last two decades as a consequence of 
increased mission-oriented activities which in large part have been supported 
by federal funds. As a consequence, there has been an emerging concern with 
the more general concept of program management. The purpose of this 
chapter is to discuss the general concept of program management, present 
characteristics of projects, discuss the general approa:hes to placement of 
projects in organizations, outline some of the problem? faced by the project 
manager, and to indicate some of the factors or conditions which make for 
a productive project management. 

What do we mean by the concept of project management? An answer 
to this question might be found by examining defir.itions which have been 
presented by persons who have done some writing ir this field. Baumgartner 3 
defines project management to . . consist of the actions involved in pro- 
ducing project deliverable items on time, within the contemplated cost, with 
the required reliability of performance . . . Clcland 3 defines project man- 
agement as . . the means of managing a large aggregation of resources 
across functional and organizational lines of authority.” Gaddis* discusses 
project management in terms of the project manager's ability to use the 
brain power of professionals and specialists in the creation of a product 
from its initial conceptualization through testing through production. The 
above definitions can be summarized to indicate that the project manager’s 
principal role is to produce a product by integrating professional persons 
into a team operating within time, cost, and performance parameters with 
that team operating within some lines of organizational responsibilities and 
authority. 

Such definitions, while useful to provide a general orientation, are not 
sufficient by themselves to fully explain the nature of project management 



’Preliminary draft of a chapter for a book. Educational Project Manage ■ 
ment, to be published by Charles E. Merrill Co.. Reproduction is prohibited 
without permission of author. 

•John S. Baumgartner. Project Management (Homewood, Illinois: Richard 
D. Irwin, Inc., 1963). 

•David I. Cleland, ‘ Why Project Management?’* Business fforuons, VII 
< 1964), 81-86. 

*PaulO-Gaddis,"The Project Manager." Han-arcf Business Review, XXVII 
(June. 1959), 189-97. 
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